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Abstract 


A  Defence  Research  and  Development  Canada  (DRDC)  -  Atlantic  research  program  is  in 
progress  investigating  human  performance  issues  related  to  the  use  of  automated  visual  alerts 
under  high  workload  conditions.  Previous  research  comparing  performance  of  two  distinct  styles 
of  visual  alerts  -  ‘flashing  border’  versus  ‘static  sidebar’  was  completed  by  CAE  Professional 
Services  (CAE  PS)  for  DRDC  Atlantic.  CAE  PS  was  then  contracted  to  modify  the  existing 
experimental  software  and  assist  in  the  resulting  pilot  and  primary  studies  (Study  2).  Specifically 
the  software  was  modified  to  include  static  and  flashing  variables  for  both  alert  types  and  to 
modify  the  design  and  software  to  ensure  that  participants  are  visually  attending  to  all  three 
screens.  The  current  report  documents  the  work  conducted  by  CAE  PS  for  Study  2  and  includes  a 
review  of  the  experiment  procedure,  setup,  data  collection  method,  and  observations  made  during 
the  pilot  and  primary  studies. 


Resume 


Un  programme  de  recherche  de  Recherche  et  developpement  pour  la  defense  Canada  (RDDC)  - 
Atlantique  enquete  presentement  sur  des  questions  de  rendement  humain  reliees  a  l’utilisation 
d’alertes  visuelles  automatisees  sous  une  charge  de  travail  elevee.  CAE  Services  professionnels 
(CAE  PS)  avait  precedemment  effectue  pour  RDDC  Atlantique  des  recherches  qui  comparaient  le 
rendement  de  deux  types  distincts  d’alertes  visuelles  -  bordure  clignotante/barre  d’etat.  CAE  PS 
s’est  ensuite  vu  octroyer  un  contrat  pour  modifier  le  logiciel  experimental  existant  et  contribuer 
aux  etudes  pilote  et  primaire  resultantes  (2e  etude).  Plus  precisement,  le  logiciel  et  sa  conception 
ont  ete  modifies  pour  inclure  des  variables  statiques  et  de  clignotement  pour  les  deux  types 
d’alertes  et  assurer  la  surveillance  visuelle  des  trois  ecrans  par  les  participants.  Le  present  rapport 
explique  les  travaux  effectues  par  CAE  PS  pour  la  2e  etude  et  comprend  un  examen  de  la 
methode  d’ experimentation,  de  sa  preparation,  de  la  methode  de  recueil  des  donnees  et  des 
observations  faites  pendant  les  etudes  pilote  et  primaire. 
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Executive  summary 


Evaluation  of  Visual  Alerts  in  the  Maritime  Domain:  Study  2  - 
Program  Modifications 

Shelley  Roberts,  DRDC  Atlantic  CR  2008-268;  Defence  R&D  Canada  -  Atlantic; 
February  2009. 

Introduction 

The  Halifax  Class  Frigate  operations  room  is  a  demanding,  high  intensity  environment  with 
automated  systems  implemented  to  warn  operators  of  system  and  tactical  states.  Most  of  the 
alerting  systems  are  auditory,  but,  due  to  their  persistent  and  uninformative  nature,  operators  tend 
to  turn  off  the  alerting  system  as  part  of  the  watch  changeover  procedure.  A  DRDC  Atlantic 
research  program  is  in  progress  to  explore  methods  of  enhancing  the  way  operators  are  alerted  in 
complex  high  intensity  environments,  like  frigate  operation  rooms. 

In  a  previous  study,  the  detectability  of  two  visual  alerts  was  investigated.  The  alerts  were  a 
flashing  border  surrounding  the  screen,  and  a  sidebar  located  in  the  perimeter  of  the  screen.  Since 
an  operator’s  workstation  will  constist  of  three  displays,  presenting  alerts  on  one  or  all  three 
displays  was  also  examined.  Regardless  of  the  location  of  the  alerts,  participants  responded  faster 
to  the  sidebar  compared  to  the  flashing  border,  with  the  quickest  response  time  obtained  from  the 
sidebar  alert  when  presented  across  all  three  screens  simultaneously.  The  current  project  consisted 
of  modifying  the  existing  experimental  software  and  assisting  in  the  pilot  and  primary  studies. 

Results 

The  software  was  modified  to  include  static  and  flashing  variables  for  both  alert  types  (border  and 
sidebar)  and  to  modify  the  experiment  design  and  software  to  ensure  that  participants  are  visually 
attending  to  all  three  screens.  The  current  report  documents  the  work  conducted  by  CAE  PS  and 
includes  a  review  of  the  experiment  procedure,  setup,  data  collection  method,  and  observations 
made  during  the  pilot  and  primary  studies 

Significance 

Results  from  these  modifications  will  allow  DRDC  Atlantic  to  conduct  further  experimentation 
into  whether  visual  alerts  are  a  viable  alternative  in  complex  high  intensity  environments  like  the 
Halifax  Class  Frigate  operations  room. 

Future  plans 

Future  work  will  continue  to  investigate  properties  of  visual  alerts  with  the  aim  of  providing 
operators  with  an  informed  tool  for  alerting  to  system  and  operational  states.  Research  issues  will 
include  the  following:  location  of  the  sidebar  on  the  display  and  event  based  data  capture. 
Findings  will  be  applicable  to  any  complex  multi-display  environment  where  the  need  to  alert 
operators  of  system  states  exist,  such  as  command  posts  and  operations  centers  within  the  military 
domain. 
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Introduction  ou  contexte: 

La  salle  des  operations  des  fregates  de  classe  Halifax  est  un  environnement  tres  intense  et 
exigeant,  avec  des  systemes  automatises  mis  en  place  pour  alerter  les  operateurs  de  l’etat  du 
systeme  et  de  la  situation  tactique.  La  plupart  des  systemes  d’alerte  sont  auditifs  mais,  en  raison 
de  leur  nature  persistante  et  non  informative,  les  operateurs  ont  tendance  a  neutraliser  le  systeme 
d’alerte  dans  le  cadre  du  processus  de  releve  de  quart.  Un  programme  de  recherche  de  RDDC 
Atlantique  est  en  cours  pour  explorer  les  methodes  permettant  d’ameliorer  la  maniere  d’ alerter  les 
operateurs  dans  des  environnements  tres  complexes  et  intenses,  comme  la  salle  des  operations  des 
fregates. 

Au  cours  d’une  etude  precedente,  la  detectabilite  de  deux  alertes  visuelles  avait  fait  l’objet  d’une 
recherche,  les  alertes  consistaient  en  une  bordure  clignotante  entourant  le  perimetre  de  l’ecran,  et 
en  une  barre  d’etat  laterale  placee  dans  le  perimetre  de  l’ecran.  Comme  un  poste  de  travail 
d’operateur  consiste  en  trois  ecrans  d’affichage,  la  presentation  des  alertes  sur  l’un  ou  tous  les 
trois  ecrans  a  egalement  fait  l’objet  d’un  examen.  Quel  que  soit  l’emplacement  des  alertes,  les 
participants  reagissaient  plus  rapidement  a  la  barre  laterale  par  rapport  a  la  bordure  clignotante,  le 
temps  de  reaction  le  plus  rapide  etant  obtenu  de  l’alerte  de  barre  laterale  lors  qu’elle  s’affichait 
simultanement  sur  les  trois  ecrans.  Le  projet  courant  consistait  en  la  modification  du  logiciel 
experimental  existant  et  a  la  contribution  aux  etudes  pilote  et  primaire. 

Res  u  I  tats: 

Le  logiciel  et  sa  conception  ont  ete  modifies  pour  comprendre  des  variables  statiques  et  de 
clignotement  pour  les  deux  types  d’alerte  (bordure  et  barre  laterale)  et  pour  assurer  la  surveillance 
visuelle  des  trois  ecrans  par  les  participants.  Le  present  rapport  explique  les  travaux  effectues  par 
CAE  PS  et  comprend  un  examen  de  la  methode  d’experimentation,  de  sa  preparation,  de  la 
methode  de  recueil  des  donnees  et  des  observations  faites  pendant  les  etudes  pilote  et  primaire. 

Importance: 

Les  resultats  de  ces  modifications  permettront  a  RDDC  Atlantique  de  mener  d’autres  experiences 
afm  de  determiner  si  les  alertes  visuelles  constituent  une  solution  de  rechange  valable  dans  des 
environnements  complexes  et  tres  intenses  comme  la  salle  des  operations  des  fregates  de  la  classe 
Halifax. 

Perspectives: 

Les  travaux  fiiturs  continueront  a  examiner  les  proprietes  des  alertes  visuelles,  dans  le  but  de 
foumir  aux  operateurs  un  d’alerte  specialise  sur  l’etat  du  systeme  et  la  situation  operationnelle. 
Les  sujets  de  recherche  comprendront :  l’emplacement  de  la  barre  laterale  sur  l’ecran  d’affichage 
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et  la  saisie  de  donnees  sur  des  evenements.  Les  faits  constates  seront  applicables  a  tout 
environnement  complexe  a  ecrans  d’affichage  multiples  ou  existe  le  besoin  d’alerter  les 
operateurs  sur  l’etat  d’un  systeme,  comme  les  postes  de  commandement  et  les  centres 
d’operations  dans  le  domaine  militaire. 
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1  Introduction 


1.1  Background 

The  Halifax  Class  Frigate  operations  room  is  a  demanding,  high  intensity  environment,  manned 
by  approximately  twenty  Navy  personnel,  most  of  whom  are  sensor  operators.  Automated 
systems  to  assist  operators  are  a  necessity  in  this  busy  environment  and  an  automated  auditory 
alerting  system  currently  exists  in  the  operations  room  to  warn  of  impending  system  and  tactical 
states.  However,  due  to  the  persistent  and  uninformative  nature  of  the  alerts,  operators  tend  to 
ignore  them  or  turn  off  the  alerting  system  as  part  of  the  watch  changeover  procedure. 

A  previous  study  (Study  1)  by  Defence  Research  and  Development  Canada  (DRDC)  -  Atlantic 
and  CAE  Professional  Services  (PS)  explored  methods  of  enhancing  the  way  operators  are  alerted 
in  complex  high  intensity  environments,  like  a  frigate  operations  room  [1].  This  experiment 
compared  user  response  performance  using  two  distinct  styles  of  visual  alerts  -  ‘flashing  border’ 
versus  ‘sidebar’.  The  flashing  border  style  consisted  of  a  flashing,  thick  solid  red  line  around  the 
perimeter  of  the  display  (see  Figure  1.A).  The  sidebar  alert  used  a  similar  line  style  (thick  solid 
red),  but  did  not  flash,  and  was  limited  to  a  single  vertical  line  segment  at  the  display’s  upper  left 
perimeter  (see  Figure  l.B).  Additionally,  as  the  operator’s  workstation  is  expected  to  consist  of 
three  displays,  whether  to  present  visual  alerts  on  one  or  all  of  the  displays  was  also  investigated 
in  this  study.  To  mimic  a  high  intensity  workload,  participants  were  provided  with  information 
regarding  targets  and  required  to  identify  them  as  hostile  or  neutral  depending  on  the  information 
provided.  Results  indicated  that,  regardless  of  alert  location,  participants  responded  faster  to  the 
static  sidebar  compared  to  the  flashing  border,  with  the  fastest  response  time  to  the  static  sidebar 
alert  when  it  was  presented  across  all  three  screens  at  one  time. 
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1 .2  Scope  of  Current  Work 

The  current  study  (Study  2)  is  a  continuation  of  effort  to  further  investigate  the  properties  of 
visual  alerts  with  the  aim  of  providing  operators  with  an  informed  tool  for  alerting  to  system  and 
operational  states.  Based  on  the  finding  in  previous  work,  where  static  sidebar  alerts  were 
detected  faster  than  flashing  border  alerts,  the  present  study  was  designed  to  tease  apart  the  effect 
found  by  examining  each  attribute  of  the  alert  stimuli.  Thus  the  attributes  of  alert  behaviour 
(flashing  vs  static)  as  well  as  alert  type  (border  vs  sidebar)  were  investigated. 

Specifically  CAE  PS  was  contracted  to: 

1 .  Modify  the  experiment  software  program; 

a.  Add  flashing  and  static  parameters  within  the  experimental  software  to  control 
the  behaviour  of  both  the  existing  border  and  sidebar  style  alerts,  and 

b.  Increase  the  number  of  target  types  from  2  (hostile,  neutral)  to  3  (hostile,  neutral, 
friendly). 

2.  Conduct  a  pilot  study; 

a.  Complete  any  needed  modifications  based  on  the  results  of  the  pilot  study;  and 

3.  Assist  in  running  the  primary  study. 

Tasks  for  both  the  pilot  and  primary  studies  included  briefing  participants;  administering  each 
participant  through  the  experiments;  reimbursing  participants;  and  collecting  data.  This  final 
report  provides  documentation  of  the  experimental  modifications,  procedure,  data  collection 
method,  and  any  relevant  observations  made  during  the  pilot  and  primary  studies.  All 
modification  that  resulted  in  differences  between  Study  1  [1]  and  Study  2  as  well  as  those  that 
resulted  in  differences  between  Study  2’s  pilot  and  primary  studies  are  discussed  in  this 
document. 
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2  Documented  Modifications 


2.1  Experimental  Software  Modifications 

Through  consultation  with  the  Scientific  (SA)  and  Technical  Authorities  (TA)  an  understanding 
of  the  required  modifications  to  the  existing  experimental  design  was  attained  and  a  plan  was 
developed  to  complete  these  modifications. 

Within  the  E-Prime  environment,  there  are  two  areas  that  required  changes;  the  E-Basic  script  and 
the  end-user  E-Studio  environment.  E-Basic  is  nearly  identical  to  Visual  Basic  for  Applications™ 
and  works  as  the  underlying  scripting  language  of  E-Prime.  The  graphical  representations  of 
experiments  are  prepared  in  E-Studio  and  are  automatically  compiled,  creating  a  run-time  file.  E- 
Basic  is  useful  when  the  experiment  requires  operations  that  are  not  supported  by  the  E-Studio 
end-user  visualization  tools.  The  current  study  would  account  for  a  non-standard  operation  as  it 
spans  across  three  screens  and  is  a  dynamic  and  interactive  environment  not  readily  supported  by 
the  E-Studio  environment.  The  following  sections  describe  the  changes  made  to  the  relevant  areas 
of  E-Basic  and  E-Studio. 

2.1.1  Parameters  Added  to  Software 

Parameters  for  controlling  the  flashing  and  static  behaviour  of  the  existing  border  and  sidebar 
alerts  were  added  to  the  software.  The  new  parameter  was  programmed  so  that  the  experimenter 
can  select  the  alarm  type  (static  versus  border)  and  alert  behaviour  (static  versus  flashing)  within 
the  E-Studio  environment.  To  change  the  parameters,  a  list  titled  “BlockLisf  ’  within  the  E-Studio 
file  can  be  used.  To  change  the  type  of  alert  the  parameters  titled  “AlarmType”, 
“AlarmTypeDesc”,  “AFlashCondl”  and  “AFlashCondlDesc”  are  changed.  Table  1  describes 
these  parameters.  A  screen  shot  of  the  BlockList  E-Studio  object  that  contains  these  parameters 
can  be  found  in  Figure  2. 

Table  1.  Parameters  provided  to  control  for  alarm  types.  Note  that  this  is  found  in  the  E-Studio 

end-user  object  called  “BlockList”. 


Alarm 

AlarmType 

AlarmTypeDesc 

AFlashCondl 

AFlashCondlDesc 

Static  Side  bar 

1 

Side  Bar 

0 

Static 

Static  Border 

2 

Border 

0 

Static 

Flashing  Side  bar 

1 

Side  Bar 

1 

Flashing 

Flashing  Border 

2 

Border 

1 

Flashing 

A  parameter  was  added  that  allows  the  experimenter  to  select  the  number  of  blocks  participants 
will  receive.  This  parameter  is  also  found  in  the  “BlockList”  object  within  the  end-user  E-Studio 
environment.  This  is  denoted  by  the  “weight”  column  provided  in  the  ‘BlockList’  object  (see 
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Figure  2).  In  the  example  provided  in  Figure  2  participants  would  receive  4  blocks  of  each  type  of 
alert,  a  total  of  16  blocks,  as  denoted  by  the  weighting  of  each  row  however  this  column  may  be 
modified  by  the  experimenter. 


AAS-CAE_Phase  2_Final  -  E-Studio  Professional  -  [BlockList  (List)] 


Experiment  (AAS-CAE_Phase  2_Final.  a 
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|j  InLineDedArrays 

3  InLineSDInfo 
y  InLineTargExtras 
=|  InLineTrials 

[A]  SeeExperimenter 
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Weight  |  Procedure 

AlarmType 

AlarmTypeDesc 

AFlashCondl 

AFlashCondl  Desc 

1 

4  pBlockProc 

1 

Side  Bar 
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2 
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Figure  2.  A  screen  shot  of  the  E-Studio  end  user  BlockList  object  containing  the  new  parameters 
that  control  alert  type.  The  four  parameters  that  experimenters  can  change  are  labelled  in  Table 
1  and  surrounded  with  the  red  box  in  the  current  figure. 


2.1.2  Addition  of  Target  Type  “Friendly” 

A  concern  about  the  experimental  design  emerged  based  on  observations  from,  and  discussion 
about  Study  1  [1],  This  concern  was  that  participants  could  feasibly  enter  an  answer  (e.g.  “asd”), 
followed  by  feedback  that  they  were  wrong,  and  without  consulting  the  Status  screen  again  enter 
the  alternate  answer  (“qwe”).  That  is,  the  need  to  consult  the  Status  (left-hand)  screen  to  classify 
targets  was  minimal  as  the  selection  was  binary  and  participants  could  succeed  by  guessing. 
Various  recommendations  and  decisions  were  made,  as  described  in  the  following  paragraphs. 

The  first  suggestion  to  manage  this  procedural  concern  was  that  a  third  target  classification 
‘friendly’  be  added  to  the  existing  two  targets  types,  ‘hostile’  and  ‘neutral’.  It  was  hoped  that 
including  a  third  choice  in  the  decision  making  task  would  provide  participants  with  a  stronger 
incentive  to  use  the  Status  Display  in  making  their  decision  regarding  assigning  a  category  to  a 
target  and  decrease  the  rate  of  chance  from  50%  to  33%. 

A  fourth  characteristic  of  the  targets,  called  ‘flag’  would  also  be  added  to  the  existing  three 
characteristics;  size,  speed  and  weapons.  ‘Flag’  would  have  two  options,  Canadian  or  Other. 
Alone,  ‘flag’  would  determine  whether  or  not  a  target  was  classified  as  ‘Friendly’.  A  Canadian 
flag  would  indicate  ‘Friendly’  no  matter  what  the  original  three  characteristics  dictated,  therefore 
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it  would  veto  any  other  characteristic  listed.  This  action  was  chosen  to  reduce  task  difficulty.  The 
intention  of  including  an  additional  target  type  was  not  to  increase  task  difficulty  but  merely  to 
ensure  participants  were  using  the  Status  display  to  make  target  identification  decisions.  An 
“Other”  Flag  classification  would  indicate  that  the  participant  would  need  to  refer  to  the  three 
characteristics,  size,  speed  and  weapons,  to  determine  if  the  target  was  ‘hostile’  or  ‘neutral’.  This 
modification  of  adding  another  target  type  and  category  (see  Table  3)  was  chosen  as  a  viable 
option  as  it  is  similar  to  the  paradigm  used  in  Study  1  [1]  (see  Table  2).  Because  it  could  be  quite 
easy  for  participants  to  classify  ‘friendly’  targets  it  was  also  decided  that  1/3  of  targets  would  be 
friendly.  Therefore,  although  it  would  be  easy  for  participants  to  clear  the  1/3  friendly  targets,  the 
other  2/3  of  targets  would  be  more  difficult  to  clear. 

An  effort  by  CAE  PS  was  made  to  add  the  ‘Friendly’  target  classification  and  the  ‘Flag’ 
characteristic.  However,  because  a  significant  amount  of  time  was  spent  attempting  to  incoiporate 
these  changes  to  the  E-Basic  script  the  SA  and  TA  were  notified  as  to  the  difficulties. 

Following  discussion  with  the  SA  and  TA  it  was  decided  that  the  number  of  target  types  be 
increased  from  2  (hostile,  neutral)  to  3  (hostile,  neutral,  friendly).  This  scheme  is  ver\>  similar  the 
previously  proposed  scheme  however  the  fourth  characteristic  “Flag”  was  removed  and  the 
target  classification  “friendly”  was  introduced  across  all  characteristics  (size,  speed  and 
weapons).  This  decision  was  made  in  the  hope  that  decreasing  the  chance  of  being  correct  based 
on  guessing  would  encourage  participants  to  consult  the  Status  display  prior  to  classifying  the 
selected  targets  as  ‘hostile  ’,  ‘unknown  ’  (previously  ‘neutral  ’)  or  friendly  ’.  This  is  similar  to  the 
first  suggestion,  however  a  fourth  target  type  ‘Flag’  would  not  be  implemented  and  the  number  of 
target  types  would  increase  from  2  (neutral  and  hostile)  to  3  (neutral,  hostile  and  friendly).  The 
target  characteristics  would  also  be  altered  to  incorporate  the  third  target  type  friendly  ’.  The 
target  types  and  classification  scheme  used  in  Study  1  [1]  can  be  found  in  Table  2  and  the 
proposed  target  types  and  classification  scheme  for  Study  2  can  be  found  below  in 

Table  4. 


Table  2.  Target  types  and  classification  scheme  used  in  Study  1. 


Target  Type 

Size 

Speed 

Weapons 

Response 

Neutral 

Large 

Slow 

No 

QWE 

Hostile 

Small 

Fast 

Yes 

ASD 
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Table  3.  First  proposed  target  types  and  classification  scheme. 


Target 

Type 

Size 

Speed 

Weapons 

Flag 

Response 

Neutral 

Large 

Slow 

No 

Other 

QWE 

Hostile 

Small 

Fast 

Yes 

Other 

ASD 

Friendly 

Large/Small 

Slow/Fast 

Yes/No 

Canadian 

Undecided 

Table  4.  Second  proposed  target  types  and  classification  scheme. 


Target  Type 

Size 

Speed 

Weapons 

Response 

Unknown 

(previously  know  as  “Neutral”) 

Medium 

Variable 

Possibly 

ASD 

Friendly 

Large 

Slow 

No 

QWE 

Hostile 

Small 

Fast 

Heavily 

ZXC 

Upon  further  discussion  with  the  SA  and  TA,  and  testing  the  program,  it  was  determined  that  the 
addition  of  a  third  target  type  would  not  likely  resolve  the  previous  experimental  design  concerns. 
Specifically  the  participant  could  still  feasibly  enter  one  answer  (e.g.  “asd”),  followed  by 
feedback  that  they  were  wrong,  and  without  consulting  the  Status  screen  enter  a  second  answer 
(“qwe”)  followed  by  the  third  “zxc”  if  the  second  answer  was  wrong.  To  the  reader  this  may  seem 
like  a  lengthy  pattern  of  behaviours  to  follow,  but  with  practice  this  could  lead  to  the  least  amount 
of  work  for  the  participant  to  endure.  The  issue  was  further  discussed  and  an  alternate  approach, 
that  would  address  both  concerns,  was  agreed  upon.  The  procedure  is  discussed  below. 

To  address  the  concern  that  participants  may  not  be  equally  attending  to  all  three  screens,  two 
buttons  were  added  to  the  Status  display;  one  labelled  hostile  and  one  labelled  neutral  (see  Figure 
3).  The  E-Basic  environment  was  modified  so  that  prior  to  entering  a  response  into  the  Reporting 
display,  participants  were  required  to  select  their  answer  by  mouse  clicking  on  the  correct  button 
on  the  Status  display  (see  Figure  3).  This  additional  step  ensured  that  the  Status  Display  was 
attended  to. 


6 


DRDC  Atlantic  CR  2008-268 


As  previously  discussed,  the  second  concern  was  that  participants  could  feasibly  enter  one  answer 
(e.g.  “asd”),  be  provided  feedback  that  they  were  wrong,  and  without  consulting  the  Status  screen 
again  enter  the  other  answer  (“qwe”).  This  concern  was  addressed  by  modifying  the  E-Basic 
environment  so  that  the  selected  target  was  un-selected  following  an  incorrect  answer  into  the 
Reporting  display.  Therefore,  if  participants  were  incorrect  they  would  have  to  repeat  the  entire 
categorization  task  by  re-selecting  a  target,  selecting  the  appropriate  button  on  the  Status  display 
and  then  entering  their  answer  into  the  Reporting  display.  This  modification  also  has  the  potential 
to  increase  the  amount  of  attention  the  participants  were  giving  to  each  screen. 


Figure  3.  Screen  capture  of  the  modified  Status  display.  The  red  arrows  indicate  the  “ neutral ” 

and  “hostile”  buttons. 

2.1.3  Additional  Modifications 

Additional  changes  to  the  experiment,  including  any  new  or  modified  script  files,  modules, 

functions,  subroutines,  parameters,  and  output  formats  are  listed  below: 

1.  In  Study  1  [1]  the  ratio  of  hostile  to  neutral  targets  was  configured  to  be  30:70.  Following  the 
pilot  for  Study  2  and  a  discussion  with  the  SA  and  TA  it  was  decided  that  the  ratio  of  hostile 
to  neutral  targets  be  equal  and  therefore  it  was  changed  to  50:50.  This  is  a  parameter  that  can 
be  controlled  by  the  experimenter  within  the  E-Basic  script. 

2.  Targets,  symbolized  as  yellow  triangles,  originated  in  the  periphery  of  the  tactical  display  and 
moved  toward  the  ownship,  located  at  the  centre  of  the  display  represented  by  a  light  grey 
circle  (See  Figure  4).  Participants  were  warned  that  keeping  targets  from  the  ownship  was 
critical,  because  if  a  target  reached  the  ownship,  it  would  be  destroyed.  If  a  target  (hostile  or 
neutral)  met  with  the  ownship  the  program  was  immediately  halted,  an  audio  file  of  an 
explosion  was  played,  and  a  picture  of  a  ship  being  destroyed  was  displayed  on  the  middle 
screen  for  a  pre-set  duration  (see  Figure  5).  The  session  then  started  over,  with  the  targets 
originating  in  the  periphery  of  the  tactical  display  and  moving  towards  the  ownship  again. 
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Figure  4.  Screen  capture  of  the  Tactical  (middle)  display. 


Figure  5.  Screen  capture  of  a  ‘ship  explosion 

In  Study  1  [1]  each  ‘ship  explosion’  was  displayed  for  5  seconds  (500ms).  There  was  a  possibility 
that  the  duration  of  5  seconds  could  have  interfered  with  the  number  of  alerts  that  participants’ 
experienced  in  each  block.  That  is,  if  the  ship  explosion  took  5  seconds  and  an  alert  was  displayed 
at  the  same  time,  participants  would  not  experience  that  alert.  Furthermore,  the  duration  of  a 
block  was  time-based  (3  minutes),  rather  than  being  event-based.  Consequently,  the  number  of 
alerts  presented  in  each  block  could  vary  depending  on  the  number  of  ship  explosions  that 
occurred  during  a  3  minute  period.  Following  discussion  with  the  SA,  the  ‘ship  explosion’  time 
was  reduced  from  5  seconds  to  3  seconds  (300ms),  thereby  increasing  the  likelihood  that  all  alerts 
would  be  experienced  within  a  given  block  (see  recommendations).  Decreasing  the  ship  explosion 
pause  time  also  decreased  the  amount  of  time  participants  waited  for  the  task  to  restart  -  this  was 
seen  as  a  positive  improvement  as  some  participants  in  Study  1  [1]  commented  that  the  break 
before  re-starting  block  trials  was  too  long. 
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2.2  Method  Modifications 

This  section  documents  any  modifications  made  to  the  method  including  the  participants, 
apparatus,  stimuli,  tasks,  procedure,  and  performance  measures. 

2.2.1  Participants 

The  methods  of  recruitment  and  reimbursement  used  for  the  current  pilot  and  primary  studies 
were  identical  to  those  in  Study  1  [1]  and  therefore  only  the  items  that  differed  are  explained  in 
this  section. 

Six  participants  (5  males,  1  female)  were  recruited  for  the  pilot  study  which  took  approximately 
90  minutes  to  complete.  Participants  mean  age  was  45.5  (SD=21.07)  with  a  range  of  20  to  70 
years.  One  participant  was  leftt-handed  and  all  six  had  indicated  that  they  had  previous 
experience  with  multiple  displays  while  multitasking. 

Twenty-four  participants  (15  males,  9  females)  were  recruited  for  the  primary  study  which  also 
took  approximately  90  minutes  to  complete.  Participants’  mean  age  was  41.25  (SD=12.45)  with  a 
range  of  22  to  70  years.  All  reported  normal  or  corrected-to-normal  vision,  and  79%  were  right- 
handed.  Five  of  the  24  participants  indicated  that  they  had  previous  experience  with  multiple 
displays  while  multitasking. 

2.2.2  Apparatus  and  Materials 

The  apparatus  used  for  the  current  pilot  and  primary  studies  were  identical  to  those  in  Study  1  [1]. 
The  recruitment  posters  can  be  found  in  Annex  A  and  Annex  B.  The  recruitment  poster  found  in 
Annex  A  included  experimenter  contact  information  on  tabs  that  could  be  tom  off  by  potential 
participants. 

The  consent  form,  found  in  0,  was  modified  to  include  the  latest  contact  information  and 
experimenters  involved  in  Study  2.  The  debriefing  used  in  Study  2  can  be  found  in  0. 

2.2.3  Stimuli 

The  stimuli  used  for  the  current  pilot  and  primary  studies  were  identical  to  those  in  Study  1  [1] 
except  for  the  following. 

1 .  Status  (left)  display  -  provides  information  on  a  selected  target  in  the  form  of  text. 
Information  included:  speed  (fast/slow),  size  (small/large),  and  weapons  onboard 
(yes/no).  The  current  study  was  modified  to  include  two  buttons  on  the  status  display;  one 
labelled  hostile  and  one  labelled  neutral  (see  Figure  6).  Refer  to  section  2.1.2  for  a  more 
detailed  description  of  these  changes. 


DRDC  Atlantic  CR  2008-268 


9 


Figure  6.  Screen  capture  of  the  Status  (left)  display.  Figure  A  is  a  screen  capture  of  the 
original  Status  display  and  Figure  B  is  the  modified  version  used  in  the  current  study. 

2.2.4  Task 

The  tasks  participants  were  required  to  complete  for  the  current  pilot  and  primary  studies  were 
identical  to  those  in  Study  1  [1]  with  the  exception  of  the  following. 

2.2.4. 1  Alerts 

Throughout  the  categorization  task  four  alert  types,  flashing  border,  static  border,  flashing  sidebar 
and  static  sidebar  (See  Figure  6),  were  presented  intermittently  on  one  (right,  middle  or  left)  or  all 
3  displays,  depending  on  the  experimental  condition.  Participants  were  still  required  to  attend  to 
the  alert  as  quickly  as  possible  by  pressing  the  spacebar  on  the  keyboard.  If  an  alert  was  missed  it 
would  cancel  out  after  4  seconds  and  be  recorded  as  Missed. 
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2.2. 4.2  Identify  Targets 


As  in  Study  1  [1],  participants  were  instructed  to  place  the  mouse  cursor  over  a  target  on  the 
tactical  display  to  obtain  information  related  to  the  target’s  attributes  which  would  then  appear  on 
the  status  display.  Three  attributes  were  assigned  to  each  target  (size,  speed,  and  weapons  on 
board).  Certain  attribute  values  represented  hostile  characteristics  while  others  represented  neutral 
(see  Table  5). 

Table  5.  Possible  target  attributes  presented  on  the  Status  display. 


Categorized  as  Hostile 

Categorized  as  Neutral 

Size 

Small 

Large 

Speed 

Fast 

Slow 

Weapons 

Yes 

No 

Participants  were  required  to  derive  from  the  attribute  information  a  categorization  judgement  as 
to  whether  a  target  was  neutral  or  hostile:  2  or  more  hostile  values  =  hostile;  2  or  more  neutral 
values  =  neutral.  At  this  stage  of  the  target  identification  task  there  were  no  modifications.  The 
current  study  was  modified  so  that  once  a  decision  had  been  made  the  choice  was  then  selected  on 
the  status  display  (see  Figure  6)  followed  by  reporting  the  answer  on  the  reporting  display  to  the 
right  of  the  tactical  display  as  “asd”  for  “hostile”  and  “qwe”  for  “neutral”.  The  report  was 
immediately  verified  by  the  system  as  correct  or  incorrect  and,  as  in  Study  1  [1]  ,  participants 
were  given  immediate  feedback  on  the  reporting  display.  If  the  target  was  correctly  identified  it 
would  disappear  off  the  screen  and  a  new  target  would  enter  from  the  periphery  of  the  tactical 
display,  thus  alleviating  the  number  of  targets  on  the  screen  that  were  located  close  to  the 
ownship.  However,  unlike  Study  1  [1]  ,  if  a  target  was  incorrectly  identified  the  target  would  be 
deselected  and  continue  to  move  towards  the  ownship  until  it  was  re-selected  and  correctly  tagged 
using  the  mouse,  identified,  and  reported. 

If  a  target  (hostile  or  neutral)  met  with  the  ownship,  represented  by  a  light  grey  circle  in  the 
middle  of  the  Tactical  display,  the  program  was  immediately  halted,  an  audio  file  of  a 
“explosion”  was  played,  and  a  picture  of  a  ship  being  destroyed  was  displayed  on  the  middle 
screen  for  3  seconds.  The  session  would  then  start  over,  with  the  targets  originating  in  the 
periphery  of  the  tactical  display,  moving  towards  the  ownship  again. 

2.2.5  Modifications  to  Procedure 

The  procedure  followed  for  the  current  pilot  and  primary  studies  was  identical  to  that  in  Study  1 
[1]  with  the  exception  of  the  following: 

The  practice  session  began  with  a  set  of  instructions  identical  to  those  used  in  Study  1  [1]  with  the 
addition  of  the  following: 

“Small  or  fast  targets  are  considered  hostile  because  hostile  ships  can  use  a  “swarm”  technique, 
which  requires  ships  be  manoeuvrable  and  quick.  Large  or  slow  ships  are  considered  neutral 
because  these  ships  often  pose  little  threat  to  combat  ships.  To  protect  your  own  ship  you  will 
determine  whether  targets  approaching  your  ownship  are  hostile  or  neutral.” 
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The  instructions  were  followed  by  a  practice  session  consisting  of  four  blocks.  Each  block 
consisted  of  one  of  the  four  alert  types  (flashing  border,  static  border,  flashing  sidebar,  static 
sidebar).  Within  each  block  alerts  were  displayed  16  times  across  4  locations  (4  alerts  on  each  of 
the  right,  middle,  left  and  all  three  displays).  The  order  in  which  the  alert  appeared  on  the  4 
locations  within  a  block  was  randomly  selected  via  E-Prime. 

The  main  experiment  directly  followed  the  practice  session  and  consisted  of  sixteen  blocks 
(lasting  approximately  3  minutes  each).  Each  participant  received  4  blocks  of  each  alert  type  and 
alert  behaviour  (4  flashing  sidebar  and  4  flashing  border,  4  static  sidebar  and  4  static  sidebar), 
with  the  order  of  the  blocks  controlled  for  using  a  Latin  square  design. 

2.3  Modifications  to  Data  Collection 

The  type  and  manner  in  which  data  were  collected  is  described  in  this  section.  There  are  two 
subsections,  the  first  (2.3.1),  describes  the  modifications  made  within  the  E-Studio  environment 
and  the  second  (2.3.2)  describes  the  modifications  and  observations  made  within  the  E-Basic 
environment. 

2.3.1  Data  Collection  via  E-Studio 

2. 3. 1.1  Demographics  Collection 

The  following  demographics  were  collected  via  the  E- Studio  environment  within  the 
‘Startuplnfo’  screen  (see  Figure  5)  which  can  be  altered  for  future  experiments.  All  but  “Time 
spent  reading”  was  included  in  Study  1  [  1  ]  and  therefore  will  be  discussed  in  more  detail  below. 

1.  Subject  number; 

2.  Age; 

3.  Gender; 

4.  Elandedness;  and 

5.  Time  spent  reading  per  week. 

2.3. 1.1.1  Time  Spent  Reading 

Previous  research  has  found  that  a  “good”  reader  (or  at  least  someone  who  practices  and  enjoys 
reading)  has  a  more  efficient  saccadic  movement  than  a  person  who  doesn’t  practice  reading  on  a 
regular  basis  [2].  One  might  hypothesize  that  larger  and  more  frequent  saccadic  movements 
(typical  of  a  practiced  reader)  would  perform  better  at  scanning  the  displays  for  alerts.  Therefore 
each  participant  was  asked  how  much  time  they  spent  reading  for  pleasure  on  a  weekly  basis. 
This  demographics  question  was  added  to  the  E-Studio  environment  within  the  experimental 
‘Startuplnfo’  screen  (see  Figure  7). 
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Figure  7.  Screen  capture  of  the  E-Studio  environment ’s  Startup  Info 

Because  demographics  were  collected  via  the  E-Studio  environment  the  data  are  saved  as  an  E- 
DataAid  2.0  file  for  each  participant.  The  demographics  are  the  only  data  that  are  collected  via  E- 
Studio  with  the  rest  of  the  data  being  collected  by  E-basic  and  written  to  an  external  (from  E- 
DataAid)  data  file. 

2.3.2  Data  Collection  via  E-Basic 

Although  there  are  63  variables  that  are  collected  by  the  E-Basic  script,  to  simplify  this  document 
only  the  variables  that  are  used  to  organize  and  analyze  the  data  of  interest  to  the  analyses  are 
described  below.  In  0  four  columns  are  used  to  classify  and  describe  each  variable  and  are  listed 
below: 

1.  Variable  Name:  This  is  the  name  of  the  variable,  found  in  the  first  row  of  the  data  file; 

2.  Manipulated  by  Experimenter:  This  provides  the  reader  with  information  regarding 
whether  or  not  the  variable  is  easily  changed.  Refer  to  the  last  column  titled  “Possible 
Values”  for  values  of  variables  that  can  be  manipulated; 

3.  Description:  This  column  describes  the  variable;  and 

4.  Values:  This  column  lists  the  current  value(s)  of  each  variable  (e.g.,  the  number  of 
targets  on  the  Tactical  display  at  any  one  time  is  currently  set  to  10.  The  value  can  be 
easily  changed  in  the  script  by  the  experimenter). 

2.3.2. 1  Performance  Measures 

This  section  documents  modifications  and  technical  hitches  for  the  performance  measures 
observed  during  the  pilot  and  primary  studies. 
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2. 3.2. 1.1 


Hits,  Misses  and  False  Hits 


Participants  were  required  to  press  the  spacebar  key  when  they  noticed  any  type  of  alert  which 
was  categorized  as  a  “Hit”.  If  participants  did  not  respond  to  the  alert  while  it  was  still  visible, 
this  was  categorized  as  a  “Miss”.  A  third  behaviour,  categorized  as  a  “False  Hit”  occurred  when 
participants  pressed  the  space  bar  while  there  was  no  alert  present.  In  Study  1  [1],  a  False  Hit  was 
analyzed  together  with  Misses.  It  was  the  hope  of  the  project  team  that  in  Study  2  the  number  of 
False  Hits  could  be  parsed  out  as  a  separate  performance  measure  from  the  Misses.  However,  the 
manner  in  which  data  is  collected  presently  does  not  decipher  between  a  False  Hit  and  a  Miss,  as 
they  are  each  given  the  same  value. 

With  the  aid  of  the  project  TA  a  script  was  written  to  parse  out  the  two  different  types  of 
responses.  More  specifically,  if  the  participant  responds  immediately  after  an  alert  has  left  the 
display  there  will  be  a  response  time  after  the  alarm  offset  time,  and  the  response  will  be 
classified  as  a  Miss.  However,  if  it  is  a  False  Hit  there  will  be  a  response  time  but  no  obvious 
alarm  to  link  it  to.  It  was  through  this  difference  in  the  data  files  that  False  Hits  were 
distinguished  from  Misses 

2. 3.2. 1.2  Alert  Responses 

Upon  testing  the  software  and  inspection  of  the  pilot  study  data  files  it  was  revealed  that  there 
were  two  issues.  The  first  was  that  the  Hit  data  was  not  consistently  recorded.  The  program 
collected  data  in  a  time-based  manner  in  2  second  windows  and  there  were  instances  where  a 
participant  could  respond  more  than  once  to  an  alert  within  that  2  second  time  span.  Ideally,  the 
first  response  should  be  recorded  as  a  Hit,  and  the  second  as  a  False  Hit.  However  in  Study  1  [1] 
the  second  response  would  overwrite  the  first  and  therefore  the  actual  Hit  data  was  inaccurate.  A 
review  of  the  data  from  Study  1  [1]  revealed  that  the  number  of  times  this  instance  occurred  was 
low.  Nevertheless  the  software  was  modified  for  Study  2  so  that  the  first  response  (Hit)  was 
recorded  and  not  overwritten  by  the  second  response  (False  Hit).  However,  the  limitation  of  one 
response  per  2  second  window  remained  meaning  that  responses  after  the  initial  Hit  (False  Hits) 
were  not  recorded.  This  problem  will  be  corrected  in  future  work  by  applying  more  in-depth 
modifications  to  the  software  so  that  data  is  collected  in  an  event-based  manner  instead  of  time- 
based. 

A  second  issue  with  alert  responses  and  data  collection  was  that  the  ability  to  detect  all  alerts  and 
associated  Misses  accurately.  When  alerts  were  present  on  the  screen(s)  and  not  associated  with 
any  participant  behaviour  or  alert  data,  the  data  file  sometimes  defaulted  to  a  string  of  0’s, 
resulting  in  an  exclusion  of  all  alerts  and  Misses.  Therefore,  upon  analysis  this  would  result  in  a 
lower  number  of  alerts  than  actually  experienced  by  the  participant  and  intended  by  the 
experimenter  (16  per  block),  referred  to  as  “ghost  alerts”  as  well  as  an  incorrect  frequency  of 
Misses.  This  second  issue  has  been  repaired  by  associating  which  screen  the  alert  was  displayed 
on  and  categorizing  each  instance  as  a  Miss. 
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2. 3.2. 1.3  Simultaneous  Target  Identification  and  Alert  Response 

Following  the  pilot  study  the  data  files  were  examined  and  it  was  revealed  that  when  participants 
were  simultaneously  identifying  targets,  i.e.  typing  in  “asd”  or  “qwe”  in  the  Report  display,  and 
responding  to  an  alert  at  the  same  time,  the  spacebar  response  to  the  alert  was  not  collected  by  the 
system.  This  was  corrected  by  CAE  PS  and  the  changes  are  documented  in  the  E-Basic  script  so 
that  during  the  primary  study  captured  all  the  alert  and  target  identification  responses  even  when 
they  occur  at  the  same  time. 
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3  Recommendations 


1.  One  of  the  main  findings  from  Study  1  [1]  was  that  the  sidebar  resulted  in  a  significantly 
faster  response  when  compared  to  the  flashing  border  alert.  In  this  study  the  sidebar  alert 
was  located  on  the  top  left  hand  side  of  the  screen  across  all  conditions.  One  problem 
with  presenting  the  sidebar  on  the  top-left  comer  of  the  screen  is  that  when  the  sidebar  is 
presented  on  the  left  and  right  displays  the  sidebar  is  not  equidistant  from  the  center 
display.  Therefore  it  is  recommended  that  the  location  of  the  sidebar  alert  should  be 
investigated  further  to  determine  whether  the  faster  response  is  dependent  on  the  location 
in  which  it  is  presented.  For  example  a  further  study  could  compare  the  response  times  to 
a  sidebar  presented  on  the  top  left,  top  right,  top  middle,  bottom  middle,  left  middle,  and 
right  middle  screen  locations. 

2.  In  complex  environments  attention  to  alerts  or  warnings  usually  needs  to  be  followed  by 
an  understanding  of  the  number  presented,  their  priorities,  importance,  identification  and 
localizability  [3].  To  provide  more  ecologically  valid  results  it  is  recommended  that  the 
tasks  and  paradigms  used  in  future  studies  better  reflect  real-life  operational  situations 
that  could  occur.  For  example,  there  may  be  situations  in  which  visual  and  auditory 
signals  are  given  together  rather  than  as  alternatives  to  each  other  in  alerting  human 
operators  alongside  a  cluttered  display.  For  this  type  of  setup,  signals  of  the  auditory  and 
visual  modalities  may  be  provided  separately  or  simultaneously,  synchronously  or 
asynchronously  to  operators.  For  combined  display  of  visual  and  auditory  signals,  the 
interactive  effects  are  not  yet  known  and  there  are  no  useful  research  results  or  practical 
ergonomic  guidance  available  for  use  by  industrial  designers  to  date  [4].  Additional 
ecologically  valid  paradigms  might  include  investigating  various  alert  types  and  locations 
on  multiple  operators  working  in  a  collaborative  environment  [5],  and  also  in  interacting 
with  cluttered  and  dynamic  displays  [6]. 

3.  Participants  were  to  receive  16  alerts  within  each  block.  Upon  review  of  the  data  files  it 
was  determined  that  the  number  of  alerts  experienced  within  each  block  was  not 
consistent.  This  artefact  should  be  investigated  prior  to  a  further  experiment  to  equalize 
the  number.  Although  it  was  originally  thought  that  the  ship  explosions  were  interfering, 
there  are  cases  in  the  current  pilot  study  where  there  were  no  explosions  and  there  were 
less  than  16  alerts  within  a  block.  This  may  also  be  caused  by  a  second  issue  discussed  in 
section  2. 3. 2. 1.2.  To  reiterate,  in  the  data  file  it  was  noticed  that  there  were  alerts  present 
on  the  screen(s)  but  were  not  written  to  the  data  files  correctly  resulting  in  “ghost  alerts”. 
Therefore  it  is  possible  that  participants  are  experiencing  the  intended  number  of  alerts 
per  block  but  the  data  file  is  not  collecting  this  correctly.  It  is  recommended  that  once  the 
“ghost  alerts”  are  added  to  the  data  to  be  analyzed  that  experimenters  should  ensure  that 
the  intended  number  of  alerts  per  block  is  16. 

4.  Of  particular  interest  is  an  understanding  of  what  activity  participants  are  actually 
engaged  in  while  alerts  are  presented.  For  example  are  participants  about  to  classify  a 
target,  deciding  what  target  to  select,  or  deciding  what  to  classify  the  selected  target  as.  In 
future  studies  it  might  be  useful  to  investigate  the  data  files  for  artefacts  of  participants 
selecting  a  target  (variable=”ActiveTargType”)  but  not  following  up  on  the  report  display 
immediately. 
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5.  Currently  the  software  that  collects  the  data  sends  the  information  regarding  what 
behaviours  have  occurred  every  2  seconds  to  a  data  file.  Given  this  method,  it  is  possible 
that  some  information  about  participants’  behaviour  is  lost,  as  this  method  of  data 
collection  is  time-based  not  event  driven.  Therefore  it  is  recommended  that  prior  to 
further  experimentation  a  new  method  of  data  collection  be  implemented  that  is  event 
driven  not  time-based. 
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4  Conclusions 


A  previous  study  comparing  human  behaviour  performance  to  various  types  of  visual  alerts  was 
completed  by  CAE  PS  for  DRDC  Atlantic.  CAE  PS  was  then  contracted  to  modify  the  existing 
experimental  software  and  assist  in  the  resulting  pilot  and  primary  studies.  The  current  report 
documented  the  work  conducted  by  CAE  PS  including  a  review  of  the  modifications  made  to  the 
experiment  procedure,  setup,  data  collection  method,  and  recommendations  that  emerged  through 
observations  made  during  the  pilot  and  primary  studies. 
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Annex  A  Study  2  Recruitment  Poster  with  Tabs 


DEFENCE 


RESEARCH  PARTICIPANTS  REQUESTED 
RESEARCH  &  DEVELOPMENT  CANADA  STUDY 


PARTICIPANTS: 

Males  and  females,  between  the  ages  of  1 8  and  60  years,  with  normal  or  corrected  to  normal  vision. 
OBJECTIVES: 

The  study  is  designed  to  investigate  how  a  maritime -based  high  intensity  task  impacts  detection  of  visual 
alerts,  especially  when  the  task  requires  monitoring  more  than  one  display.  The  task  involves  classifying 
and  reporting  targets  on  a  tactical  display  as  hostile  or  neutral  while  also  detecting  and  responding  to  visual 
alerts  presented  on  one  or  all  of  the  displays. 

RISKS: 

Participants  will  be  required  to  remain  vigilant  and  alert  throughout  session.  No  unusual  risks  are 
anticipated  other  than  possible  eyestrain  and  fatigue. 

BENEFITS: 

Participants  may  benefit  from  the  experience  by  gaining  knowledge  of  the  maritime  environment  and  in 
receiving  exposure  to  the  defence  related  research. 

WHEN: 

October  9-  16  2008 
WHERE: 

DRDC  Atlantic,  9  Grove  Street,  Dartmouth,  Nova  Scotia,  B2Y  3Z7 
COMPENSATION: 

Participants  will  be  compensated  according  to  DRDC  guidelines 
If  interested  please  contact: 
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Annex  B  Study  2  Recruitment  Poster 


HUMAN  FACTORS  STUDY 
PARTICIPANTS  NEEDED 


Who? 

We  require  participants  who  have  normal  or  corrected  to  normal  vision. 
We  also  require  that  you’re  18-60  years  of  age.  Participants  may  be  internal 
(to  DRDC  Atlantic)  or  external. 

What? 

You  will  be  required  to  complete  an  identification  task  spread  over  three 
computer  LCD  displays.  You  will  also  be  required  to  respond  to  mock  visual 
alarms,  also  spread  over 
three  computer  LCD 
displays 

When? 

Thursday,  October  9th  to 
Thursday,  October  16th, 

2008. 

How  long? 

Approximately  1  hour  and 
30  minutes. 

What  do  you  get  out  of  it? 

Participants  will  be  reimbursed  for  their  time  according  to  DRDC 
regulations.  You  may  also  find  benefit  in  exposure  to  DRDC  human  factors 
research. 

Who  should  you  contact  if  you’re  interested? 

Please  contact  Jeff  Beardsall  at  426-3100  x  342,  or  email  at 
ieffrey.beardsall@drdc-rddc.qc.ca 
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Annex  C  Study  2  Voluntary  Consent  Form  for 
Human  Subject  Participation 


VOLUNTARY  CONSENT  FORM  FOR  HUMAN  SUBJECT  PARTICIPATION 
Protocol  Number:  F-623  Amendment  #1 

Research  Project  Title:  Evaluating  visual  alerts  in  the  maritime  domain 

Principal  Investigator:  Dr.  Jacquelyn  Crebolder 
Co-Investigators:  Jeffrey  Beardsall 

Shelley  Roberts  -  CAE  Professional  Services 
Technical  Authority:  Don  Coady 

I, _ (name)  of _ (address  and 

phone  number)  hereby  volunteer  to  participate  as  a  subject  in  the  study, 

66  9? 

(Protocol  #F623  Amendment  #1).  I  have  read  the  information  package  on  the  research  protocol, 
and  have  had  the  opportunity  to  ask  questions  of  the  Investigator(s).  All  of  my  questions 
concerning  this  study  have  been  fully  answered  to  my  satisfaction.  However,  I  may  obtain 
additional  information  about  the  research  project  and  have  any  questions  about  this  study 
answered  by  contacting  Dr.  Jacqui  Crebolder  (902)  426-3100  x296. 

I  have  been  told  that  I  will  be  asked  to  participate  in  one  session  totaling  approximately  one  hour 
and  30  minutes  in  duration. 

The  task  is  a  computer-based  task  that  consists  of  monitoring  a  central  display  for  targets, 
gathering  information  on  those  targets  from  a  second  display,  and  reporting  the  nature  of  those 
targets  on  a  third  display. 

I  have  been  told  that  the  principal  risks  of  the  research  protocol  are:  possible  minor  eyestrain  and 
fatigue.  Also,  I  acknowledge  that  my  participation  in  this  study,  or  indeed  any  research,  may 
involve  risks  that  are  currently  unforeseen  by  DRDC  Atlantic. 

For  Canadian  Forces  (CF)  members  only:  I  understand  that  I  am  considered  to  be  on  duty  for 
disciplinary,  administrative  and  Pension  Act  purposes  during  my  participation  in  this  experiment 
and  I  understand  that  in  the  unlikely  event  that  my  participation  in  this  study  results  in  a  medical 
condition  rendering  me  unfit  for  service,  I  may  be  released  from  the  CF  and  my  military  benefits 
apply.  This  duty  status  has  no  effect  on  my  right  to  withdraw  from  the  experiment  at  any  time  I 
wish  and  I  understand  that  no  action  will  be  taken  against  me  for  exercising  this  right. 

I  have  been  advised  that  the  experimental  data  concerning  me  will  be  treated  as  confidential 
(‘Protected  B’  IAW  CF  Security  Requirements),  and  not  revealed  to  anyone  other  than  the  DRDC 
Atlantic  Investigator(s)  or  external  investigators  from  the  sponsoring  agency  without  my  consent 
except  as  data  unidentified  as  to  source.  Also,  I  understand  that  my  name  will  not  be  identified  or 
attached  in  any  manner  to  any  publication  arising  from  this  study.  Moreover,  should  it  be 
required,  I  agree  to  allow  the  experimental  data  to  be  reviewed  by  an  internal  or  external  audit 
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committee  with  the  understanding  that  any  summary  information  resulting  from  such  a  review 
will  not  identify  me  personally. 

I  understand  that  I  am  free  to  refuse  to  participate  and  may  withdraw  my  consent  without 
prejudice  or  hard  feelings  at  any  time.  Should  I  withdraw  my  consent,  my  participation  as  a 
subject  will  cease  immediately,  unless  the  Investigator(s)  determine  that  such  action  would  be 
dangerous  or  impossible  (in  which  case  my  participation  will  cease  as  soon  as  it  is  safe  to  do  so). 

I  also  understand  that  the  Investigator(s),  their  designate,  or  the  physician(s)  responsible  for  the 
research  project  may  terminate  my  participation  at  any  time,  regardless  of  my  wishes. 

I  have  been  informed  that  the  research  findings  resulting  from  my  participation  in  this  research 
project  may  be  used  for  commercialization  purposes. 

I  understand  that  for  my  participation  in  this  research  project,  I  am  entitled  to  remuneration  in  the 
form  of  a  stress  allowance  in  the  amount  of  $  1 6.68  (plus  $  1 5.00  for  travel  if  I  am  coming  from 
outside  DR  DC  Atlantic)  for  completing  one  session. 

Stress  remuneration  is  taxable.  T4A  slips  are  issued  only  for  amounts  in  excess  of  $500.00  paid 
during  a  year. 

Secondary  Use  of  Data:  I  consent/do  not  consent  (delete  as  appropriate)  to  the  use  of  this  study’s 
experimental  data  involving  me  in  unidentified  form  in  future  related  studies  provided  review  and 
approval  have  been  given  by  DRDC  HREC. 

Volunteer’s  Signature: _ Date: _ 

I  have  informed  the  Principal  Investigator  that  I  am  currently  a  subject  in  the  following  other 

DRDC  Atlantic  research  project(s): _ 

(cite  Protocol  Number(s)  and  associated  Principal  Investigator(s)),  and  that  I  am  participating  as  a 
subject  in  the  following  research  project(s)  at  institutions  other  than  DRDC  Atlantic: 
_ (cite  name(s)  of  institution(s)) 

I  understand  that  by  signing  this  consent  form  I  have  not  waived  any  legal  rights  I  may  have  as  a 
result  of  any  harm  to  me  occasioned  by  my  participation  in  this  research  project  beyond  all  risks  I 
have  assumed. 


Volunteer’s  Name  _ 

Signature: _  Date: 

Name  of  Witness  to  Signature: _ 

Signature: _  Date: 

Section  Head/Commanding  Officer’s  Signature: _ 

(see  Notes  blow) 

CO’s  Unit: 
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Principal  Investigator: _ 

Signature: _  Date: _ 

Notes: 

For  other  military  personnel:  All  other  military  personnel  must  obtain  their  Commanding 
Officer’s  signature  designating  approval  to  participate  in  this  research  project. 

For  civilian  personnel  at  DRDC  Atlantic:  Signature  of  Section  Head  is  required  designating  that 
volunteer  subject  is  considered  to  be  at  work  and  that  approval  has  been  given  to  participate  in 
this  research  project. 


FOR  SUBJECT  ENQUIRY  IF  REQUIRED: 

Should  I  have  any  questions  or  concern  regarding  this  project  before,  during,  or  after 
participation,  I  understand  that  I  am  encouraged  to  contact  Defence  R&D  Canada  -  Atlantic 
(DRDC  Atlantic),  9  Grove  Street,  Dartmouth,  Nova  Scotia,  B2Y  3Z7.  This  contact  can  be  made 
by  surface  mail  at  this  address  or  in  person,  by  phone  or  e-mail,  to  any  of  the  DRDC  Atlantic 
numbers  and  addresses  listed  below: 

Principle  Investigator  or  Principal  DRDC  Atlantic  Investigator: 

Dr.  Jacquelyn  Crebolder,  (902)  426-3100  x296,  jacqui.crebolder@drdc-rddc.gc.ca 

Or  to  the  Chair  of  the  DRDC  Human  Research  Ethics  Committee  (HREC) 

Dr.  Jack  Landolt,  (416)  635-2120,  jack.landolt@drdc-rddc.gc.ca 


I  understand  that  I  will  be  given  a  copy  of  this  consent  form  so  that  I  may  contact  any  of  the 
above-mentioned  individuals  at  some  time  in  the  future  should  that  be  required. 
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Annex  D  Study  2  Debriefing  Form 


A  Study  Investigating  Alerts  and  Multiple  Displays  in  the  Halifax  Class  Frigate. 

The  operations  room  in  a  frigate  is  a  busy  environment  where  sensor  information,  such  as  radar 
and  sonar  data,  is  translated  into  a  meaningful  picture  so  that  ship  commanders  can  understand 
what  is  going  on  above  water,  under  water,  and  in  the  air  around  them.  The  operations  room  is  a 
demanding,  high  intensity  environment,  manned  by  about  twenty  Navy  personnel,  most  of  whom 
are  sensor  operators.  Automated  systems  to  assist  operators  are  a  necessity  in  this  busy 
environment  and  an  automated  auditory  alerting  system  is  currently  in  place  in  the  operations 
room  to  warn  of  impending  system  and  tactical  states.  However,  due  to  the  persistent  and 
uninformative  nature  of  the  alerts  operators  tend  to  ignore  or  turn  off  the  alerting  system  as  soon 
as  they  come  on  duty.  The  Halifax  Class  Frigate  is  undergoing  a  complete  modernization  upgrade 
and  the  current  study  is  designed  to  explore  methods  of  enhancing  the  way  operators  are  alerted 
in  the  new  operations  room. 

Subject  matter  experts  have  suggested  static  or  flashing  borders  around  the  operator’s  computer 
screen  as  an  alternative  to  auditory  alarms.  Another  common  visual  alerting  technique  used  in 
other  environments  is  a  static  or  flashing  sidebar,  located  in  the  perimeter  of  the  screen.  The 
border  and  sidebar  are  two  alerting  techniques  that  were  investigated  in  this  study.  The  study  also 
investigated  the  behaviours  of  the  alerts  (static  or  flashing).  Additionally,  since  the  operator’s 
workstation  is  expected  to  consist  of  three  displays,  whether  to  present  visual  alerts  on  one  or  all 
of  the  displays  was  also  of  interest. 

One  of  these  alerts  may  be  easier  to  detect  than  another  and  data  was  collected  on  how  fast  each 
was  detected  while  you  were  performing  the  categorization  task.  We  also  measured  response  time 
to  the  alerts  as  a  function  of  the  display  the  alert  appeared  on  to  see  whether  it  is  worth  showing 
the  alerts  on  all  the  displays  or  whether  one  display  would  be  ample.  Additionally,  response  times 
and  accuracy  in  the  categorization  task  were  recorded  as  a  function  of  the  alert  behaviour  (static 
or  flashing).  Results  from  this  study  will  provide  insight  into  whether  visual  alerts  are  a  viable 
alternative  in  the  Halifax  Class  Frigate  operations  room  and  the  findings  will  provide  an  initial 
step  in  determining  what  types  of  alerting  techniques  might  be  most  suitable. 

Thank  you  for  your  participation  in  this  study.  We  appreciate  your  time  and  effort! 
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Annex  E  Study  2  Participant  Instructions  and 
Scenario 


Prior  to  reading  the  on  screen  instructions  participants  were  read  the  following: 

“You  are  a  control  operator  on  a  Navy  ship.  Your  primary 
responsibility  is  to  attend  and  respond  to  visual  alerts  that  appear  on 
your  displays.  Y our  secondary  task  is  to  determine  whether  targets 
approaching  your  ownship  are  neutral  or  hostile. 

I  will  have  you  read  the  first  screen  for  instruction  and  they  we  will 
continue...” 

Participants  then  read  the  following  on  the  middle  screen: 

“There  are  two  types  of  visual  alerts  that  will  appear  on  your  displays; 
a  red  bar  that  will  always  appear  on  the  left  top  area  of  the  display  and 
a  red  border  that  surrounds  the  periphery  of  your  screen.  Only  one  type 
of  display  will  appear  at  a  time.  The  displays  may  appear  on  the  left 
display,  middle  display,  right  display  or  all  three  at  once. 

When  you  hit  the  space  bar  the  alert  will  disappear  immediately. 

Alerts,  if  not  responded  to  will  disappear  after  a  short  duration. 

Just  to  remind  you  -  it  is  your  primary  task  to  respond  to  alerts. 

Any  target,  whether  it  is  neutral  or  hostile  will  destroy  your  ownship  if 
it  reaches  the  center  of  the  screen.  You  do  not  have  to  move  the  cursor 
inside  the  textbox  to  enter  your  answer  -  you  may  just  type.  However, 
you  will  need  to  move  the  cursor  over  the  enter  button  on  the  right 
screen  and  click  enter  with  the  mouse.  You  are  not  able  to  enter  your 
answer  using  the  “enter  button”  on  the  keyboard. 

To  summarize:  Responding  to  alerts  is  your  primary  task,  your 
secondary  task  is  to  identify  whether  approaching  targets  are  neutral  or 
hostile. 

Do  you  have  any  questions?” 


DRDC  Atlantic  CR  2008-268 


33 


This  page  intentionally  left  blank. 


34 


DRDC  Atlantic  CR  2008-268 


Annex  F  Variables  Used  in  E-Basic 


Variable 

Name 

Manipulated 

by 

Experimenter 

Description 

Values 

Experiment 

Yes  (via  E- 
studio) 

E-prime  Experiment  name. 

Any 

Subject 

Yes  (via  E- 
studio) 

Subject  number 

Any 

Note:  0  is  saved  for  testing  E-prime  files 
without  data  collection. 

Block 

Yes 

Block  number 

1  to  Block  n 

Trial 

No 

Trial  number.  This  number  represents  every  2 
seconds  of  the  180s  block. 

1-90 

BlockDurnSec 

Yes 

Block  Duration  (in  seconds)  Note:  this  is 
approximate,  the  program  calculates  the  number  of 
"trials"  based  on  the  block  duration  and  rounds  that 
number;  also,  the  actual  trial  time  is  usually  greater 
than  the  expected  trial  time 

180 

TotalTargets 

Yes 

Total  targets  on  screen  at  one  time,  on  screen  at 
one  time 

10 

TotalHTargets 

Yes 

Total  hostile  targets 

5 

TotaINTargets 

No 

Total  neutral  targets  (total  targets  -  hostile  targets) 

5 

Hostlnst 

Yes 

Number  of  hostile  instances  to  deem  a  target  hostile 
(i.e.,  if  hi=2  it  takes  2/3  or  more  hostile  instances  for 
a  target  to  be  considered  hostile) 

2 

TrialOnset 

No 

Trial  Onset  Time  (to  determine  actual  trial  duration 
as  the  BlockDurnSec  is  set  to  be  approximately 

180) 

Should  increase  by  approximately  2000ms 
(2seconds)  between  each  trial. 

TrialOffset 

No 

Trial  Offset  Time  (to  determine  actual  trial  duration); 
Note:  this  does  not  include  the  time  it  takes  to  write 
data  to  the  data  file 

Should  increase  by  approximately  2000ms 
(2seconds)  from  the  TrialOnset  variable. 
TrialOffset  should  =  the  TrialOnset  time  of  the 
following  trial. 

TrialDurn 

No 

Trial  Actual  Duration  (TrialOffset  -  TrialOnset) 

TrialOnset-TrialOffset=approximately  2000ms 
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TotAIrms 

Yes 

Total  number  of  alarms 

16:  Manipulated  in  the  E-Studio  environment. 
(See  Figure  2) 

AlrmFDur 

Yes 

Alarm  flashing  duration  (in  milliseconds) 

300ms 

AlrmDur 

Yes 

Alarm  duration  (in  milliseconds) 

4000ms 

AlrmFlash 

Yes  (via  E- 
studio) 

Set  whether  border  alarms  are  flashing 

This  is  an  old  variable  and  not  in  use  for  the 
current  study  (see  Table  1  to  control  alarm 
types): 

1=Yes 

0=No 

AlrmCond 

Yes  (via  E- 
studio) 

Alarm  condition 

This  is  an  old  variable  and  not  in  use  for  the 
current  study  (see  Table  1  to  control  alarm 
types): 

1=alarm  type  is  the  same  within  a  block 
2=alarm  type  varies  within  a  block 

AlrmCColour 

Yes 

Alarm  colour 

Currently  set  to  255,  0,  0 

If  changing,  refer  to  http://www.web- 
source.net/21 6_color_chart.htm 

AlrmBarWidth 

Yes 

Alarm  Width  (Side  Bars) 

Shouldn't  be  larger  than  100  pixels 

AlrmBorWidth 

Yes 

Alarm  Width  (Borders) 

Shouldn't  be  larger  than  100  pixels 

AlrmBarLength 

Yes 

Alarm  Length  (Side  Bars) 

TotAIrmSBar 

Yes  (via  E- 
studio) 

Number  of  side  bar  alarms  set  manually 

This  is  an  old  variable  and  not  in  use  for  the 
current  study.  See  Table  1  to  control  number 
of  alarms  presented. 

TotAIrmBord 

No  (via  E- 
studio) 

Number  of  border  alarms  set  manually 

This  is  an  old  variable  and  not  in  use  for  the 
current  study.  See  Table  1  to  control  number 
of  alarms  presented. 

AlrmExpOnset 

No 

Alarm  expected  onset  time  (in  milliseconds) 

The  time  the  alarm  has  been  designated  to 
appear  on  the  display(s) 

AlrmOnset 

No 

Alarm  onset  time  (in  milliseconds) 

The  actual  time  the  alarm  appears  on  the 
display(s).  Should  be  identical  to 
AlrmExpOnset 

AlrmExpOffset 

No 

Alarm  expected  offset  time  (in  milliseconds) 

The  time  the  alarm  has  been  designated  to 
disappear  from  the  display(s)  This  value 
should  be  approximately  4000ms  later  than 
the  AlrmExpOnset  and  AlrmOnset. 
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AlrmOffset 


No 


Alarm  offset  time  (in  milliseconds) 


AlrmType  Yes  (via  E- 

studio) 


AlrmCurrMon 

No 

AlrmRespTime 

No 

AlrmRT 

No 

Alrm  Accuracy 

No 

ActiveTargType  No 


Alarm  type 

Alarm  current  screen 

Alarm  response  time 
Alarm  reaction  time 
Alarm  Accuracy 


Active  target  type  selected 
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This  is  the  actual  time  the  alarm  disappears 
from  the  display(s). 

This  value  is  based  on  the  alarm  disappearing 
after  the  time  has  run  out,  not  when  the  alarm 
has  disappeared  due  to  a  response  from  the 
participant. 

This  is  an  old  variable  and  not  in  use  for  the 
current  study.  See  Table  1  to  control  for  alarm 
type. 

1  =  left/status 

2  =  middle/tactical 

3  =  right/report 

4  =  all) 

Clock.Read  at  time  participant  hits  spacebar 
AlarmRespTime  -  AlarmOnsetTime 

1.  alarm  accuracy  =  2 

a.  If  it  is  a  False  Hit,  i.e.  participant 
responds  before  or  after  an  alarm,  there  will 
be  a  response  time  but  no  obvious  alarm  to 
link  it  to. 

b.  If  a  Miss  (response  after  alarm  has  left 
the  display):  there  will  be  a  response  time 
after  the  alarm  offset  time 

2.  alarm  accuracy  =  1 

a.  If  it  is  a  Hit,  i.e.,  participant  responds 
while  alarm  is  on  display.  There  will  be  a 
response  time,  an  alarm  onset  time  but  no 
alarm  offset  time. 

3.  alarm  accuracy  =  0 

a.  If  there  is  no  response  to  an  alarm. 
There  will  be  an  alarm  onset  and  offset  time 
but  no  response  time, 
qwe  =  neutral 
asd  =  hostile 

In  the  future  this  might  be  useful  to  see 
examples  of  participant  clicking  on  a  target 
but  not  directly  following  this  action  with  an 
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answer  on  the  report  display.  See 
recommendations. 


RDResp 

No 

Report  display  response 

qwe  =  neutral 
asd  =  hostile 

RDCorrResp 

No 

Report  display  correct  response 

qwe  =  neutral 
asd  =  hostile 

RDAccuracy 

No 

Report  display  accuracy 

1=correct 

2=incorrect 

Expl 

No 

Ship  Explosion 

0=no  ship  explosion 
1=ship  explosion 
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List  of  symbols/abbreviations/acronyms/initialisms 


CAE  PS 
DRDC 
SA 
TA 


CAE  Professional  Services 
Defence  Research  &  Development  Canada 
Scientific  Authority 
Technical  Authority 
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